ADA102208 


NPS55-81-013 


NAVAL  POSTGRADUATE  SCHOOL 

Monterey,  California 


N^.%  ^u 


&  LONGITUDINAL  STUDY  OF 
COMPUTER  ^OICE RECOGNITION 
PERFORMANCE  AND  VOCABULARY  SIZE. 


Gary  K.y^oock 

Ulh**^  H> ]  3  y 


Approved  for  public  release;  distribution  unlimited. 
Prepared  for: 

Naval  Electronic  Systems  Command  ^ 

Washington,  D.C.  20360  ^ 


/  7. 


t# 


NAVAL  POSTGRADUATE  SCHOOL 
MONTEREY,  CALIFORNIA 


Rear  Admiral  J.  J.  Ekelund  D.  A.  Schrady 

Superintendent  Acting  Provost 


This  investigation  was  sponsored  by  Mr.  Frank  Deckel man,  NAVELEX,  Code  330. 
The  work  was  performed  by  the  author  at  the  Naval  Postgraduate  School , 
Monterey,  California. 


Reviewed  by 


Kneale  T.  Marshall,  Ctratrran 
Department  of  Operations  Research 


Gary  K.  Poock,  Professor 
Department  of  Operations  Research 


Released  by: 


Dean  of  Research 


Unclassified 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  (Whm  Dmlm  Bnftmd) 


REPORT  DOCUMENTATION  PAGE 


!■  i  rgrawn 


NPS55-81-013 


♦.  TITLE  (and  Submit) 


A  Longitudinal  Study  of  Computer  Voice 
Recognition  Performance  and  Vocabulary  Size 


7.  AUTHOAO) 


Gary  K.  Poock 


9.  PERFORMING  ORGANIZATION  NAME  AND  AODRESS 

Naval  Postgraduate  School 
Monterey,  CA.  93940 


II.  CONTROLLING  OFFICE  NAME  ANO  ADDRESS 

Naval  Postgraduate  School 
Monterey,  CA.  93940 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


S.  RECIPIENT'S  CATALOG  HUMMER 


*•  TYPE  OF  REPORT  *  PERIOD  COVERED 

Technical 


6.  PERFORMING  ORO.  REPORT  NUMBER 


•.  CONTRACT  OR  GRANT  NUMBER^) 


10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  *  WORK  UNIT  NUMBERS 


N0003981WR0X008 


12  WWW 


13.  NUMBER  OF  PAGES 

39 


MONITORING  AGENCY  NAME  A  ADDRESSfl/  dllimrmit  from  Controlling  Olllcm)  IS.  SECURITY  CLASS,  (ol  thlm  nport) 

Naval  Electronic  Systems  Command  Unclassified 

Washington,  D.C.  20360 


mm 


1«.  DISTRIBUTION  STATEMENT  (ol  thle  Report) 


Approved  for  public  release;  distribution  unlimited. 


17.  DISTRIBUTION  STATEMENT  (ol  thm  ebetrect  entered  In  Block  20,  II  different  from  Report) 


It.  KEY  WORDS  (Continue  on  i 


elde  II  neceeeery  end  Identify  by  block  number) 


VTAG,  Voice  Recognition,  Automatic  Word  Recognition,  Longitudinal  Study, 
Vocabulary  Size,  Joint  -  Combined  Reference  Patterns 


ABSTRACT  (Continue  on  reeeree  elde  II  neceeeery  end  Identify  by  block  number) 

This  research  examined  voice  recognition  performance  as  a  function  of 
time  and  showed  no  decrement  in  performance  after  21  weeks.  In  addition, 
vocabulary  sizes  up  to  240  utterances  showed  stable  performance.  Two 
people  also  combined  their  voice  reference  patterns  and  were  then  able 
to  achieve  an  error  rate  of  less  than  2%  when  either  person  spoke  to  the 
speaker-dependent  voice  recognition  unit. 


form  1471 

I  JAN  7»  WJ 


EDITION  OF  1  NOV  SS  IS  OBSOLETE 
S/N  0  10  3*0 14*  660 1  I 


Unclassified 


SECURITY  CLASSIFICATION  OF  THIS  PAOC  (Whm  Data  EHIaraag 


FOREWARD 


This  investigation  was  sponsored  by  Mr.  Frank  Deckelman,  NAVELEX, 

Code  330.  The  work  was  performed  by  the  author  at  the  Naval  Postgraduate 
School,  Monterey,  California. 

This  report  is  one  of  a  series  concerned  with  the  possible 
applications  of  using  voice  recognition  technology  in  command  and  control 
tasks. 

I  am  indebted  to  the  following  individuals  for  their  patience  and 
cooperation  in  working  with  me  for  over  5  months  on  this  project: 

Captain  John  Armstrong,  Canadian  Armed  Forces;  Lt.  Cdr.  Pam  Batchellor, 

USN;  Lt.  Kent  Bien,  USN;  Lt.  Col.  Paul  Bragaw,  USAF;  Capt.  Greg  Jay, 

USAF;  Prof.  Doug  Neil,  Naval  Postgraduate  School;  and  Lt.  Cdr.  Ellen  Roland,  USN. 


Accession  For 

NTIS  GRA&I 
DTIC  TAB 
Unannounced 
Justif le&t i on 


By_. 


Dio' ribut i on/ 
Avu .  i  \v.  „i  i  t 


n 

n 


i 


TABLE  OF  CONTENTS 


I.  EXECUTIVE  SUMMARY . 1 

II.  INTRODUCTION-OBJECTIVE  .  3 

III.  SUBJECTS .  3 

IV.  INITIAL  TRAINING  .  3 

V.  THE  VOCABULARY . 4 

VI.  PROCEDURE  AND  DATA  COLLECTED . 4 

VII.  RESULTS . 6 

VIII.  CONCLUSIONS . 17 

APPENDIX  A . 18 

APPENDIX  B . 19 


ii 


1 


LIST  OF  FIGURES 

Page 


FIGURE  1 .  Error  Rate  Versus  Weeks  .  8 

FIGURE  2.  Error  Rate  Versus  Vocabulary  Size  .  10 

FIGURE  3.  Error  Rate  Versus  #  Syllables  .  12 

FIGURE  4.  Recognition  Performance  by  Subject . 13 

FIGURE  5.  Performance  by  Week  for  Individual  Subjects  .  14 

FIGURE  6.  Performance  by  Week  for  Individual  Subjects  .  15 

FIGURE  7.  Performance  by  Week  for  Two  Subjects  Using  Combined 

Voice  Reference  Patterns . 16 


iii 


LIST  OF  TABLES 


I.  EXECUTIVE  SUMMARY 


This  paper  describes  an  experiment  based  on  94,560  voice  utterances 
which  examined  the  stability  of  voice  recognition  performance  over  a 
period  of  21  weeks.  The  purpose  was  to  determine  how  consistently  the 
voice  reference  patterns  would  work  and  thus  indicate  how  much  re-training 
might  be  needed  over  extended  periods  of  time.  Recognition  performance  as 
a  function  of  vocabulary  size  up  to  240  utterances  was  also  examined.  In 
addition,  a  small  effort  was  undertaken  to  examine  recognition  performance 
when  2  people  combined  their  training  passes,  i.e.  if  the  voice  recognition 
equipment  required  X  training  passes  for  each  utterance,  how  well  would 
the  recognizer  work  if  one  person  provided  half  of  the  utterances  and 
another  person  provided  the  other  half  of  the  utterances? 

Eight  subjects  participated  in  the  experiment  in  which  the  voice 
recognizer  was  tested  by  having  each  subject  repeat  480  utterances  to  the 
recognizer  each  week  for  21  weeks  in  a  relatively  noise  free  environment. 

Voice  patterns  were  not  allowed  to  be  retrained  or  "fine  tuned" 
during  the  study. 

In  addition,  a  male  and  a  female  made  a  joint  set  of  reference  patterns 
with  half  the  training  passes  for  each  utterance  provided  by  the  female  and 
half  by  the  male. 

Longitudinal  results  showed  performance  was  statistically  stable  over 
the  21  weeks  with  no  serious  degradation  occurring  as  time  elapsed.  For 
the  8  subjects  who  tested  the  recognizer  against  their  own  voice  patterns, 
the  recognition  error  rate  over  21  weeks  was  2.07%  including  both  mis- 
recognitions  and  rejections.  In  terms  of  mis-recognitions  only,  the  error 
rate  was  1.65%.  In  addition,  it  should  be  noted  that  two  people  (1  male 
and  1  female)  accounted  for  57.37%  of  these  errors. 


With  respect  to  vocabulary  size,  analysis  showed  no  statistical  dif¬ 
ferences  in  performance  when  the  vocabulary  size  was  20,  40,  60,  80,  100, 
120,  140,  160,  180,  200,  220  or  240  utterances. 

The  two  subjects  who  combined  their  voice  reference  patterns  had  an 
average  error  rate  of  1.87%  with  only  6  rejects  in  total  between  them. 

This  small  investigation  into  the  joint  reference  pattern  concept 
suggests  that  if  two  persons  are  on  a  job  and  one  is  unable  to  continue, 
the  other  person  could  step  in  and  start  using  the  same  voice  patterns, 
even  though  the  recognizer  is  a  speaker  dependent  system. 

In  summary,  voice  recognition  performance  was: 

1)  Stable  over  time. 

2)  Stable  over  vocabulary  size. 

3)  Quite  good  when  two  people  combined  their  voice 
reference  patterns  in  a  speaker-dependent  voice 
recognition  system. 


II.  INTRODUCTION-OBJECTIVE 

This  paper  describes  an  experiment  whose  purpose  was  to  examine  voice 
recognition  performance  as  a  function  of  time  and  vocabulary  size.  In 
addition,  a  small  sample  was  taken  to  examine  the  possible  use  of  joint 
training  patterns  by  two  people.  Previous  research  by  others  working  for 
this  writer  is  shown  in  Appendix  A.  The  results  of  some  of  these  studies 
suggested  voice  input  was  a  potential  way  for  improving  job  performance  in 
various  applications.  Many  of  these  studies  were  of  a  relatively  short 
duration  of  several  weeks  and  it  was  intriguing  to  know  if  the  results 
would  have  held  up  over  a  longer  period  of  time.  Hence  this  study. 

III.  SUBJECTS 

Six  military  officers  and  two  civilians  participated  in  the  longitudinal 
study  over  21  weeks.  Two  of  the  officers  were  female.  All  subjects  were 
between  20  and  40  years  old.  Their  experience  with  voice  recognition 
equipment  was  limited  to  about  a  month's  previous  experience  with  voice 
recognition  systems,  except  for  1  male  and  1  female  who  had  a  year  or  more 
of  experience.  Military  ranks  ranged  from  Lieutenant  to  Lieutenant 
Commander  in  the  Navy  and  Captain  to  Lieutenant  Colonel  in  the  Air  Force. 

IV.  INITIAL  TRAINING 

The  eight  subjects  individually  trained  a  Threshold  Technology,  Inc., 
Model  T600  voice  recognition  system  on  a  vocabulary  of  240  utterances. 

After  the  vocabulary  was  trained,  each  utterance  was  repeated  three  times 
with  the  vocabulary  wide  open  with  no  branching  or  structuring.  If  the 
voice  recognition  system  correctly  recognized  the  utterance  2  out  of  3 
times,  the  training  pattern  was  considered  valid...  it  was  retrained  if 


it  didn't  meet  this  criteria  and  tested  again  until  successful  validation 
was  acquired.  The  T600  initially  requires  10  training  passes  which  seems 
to  provide  quite  good  reference  patterns.  Less  than  2%  of  the  initially 
trained  utterances  even  needed  to  be  retrained  to  meet  the  validation 
criteria. 

Upon  successful  training,  each  subject's  voice  patterns  were  recorded 
on  individual  tape  cassettes  using  the  tape  recorder  of  the  T600.  The 
training  patterns  were  not  allowed  to  be  altered  for  the  next  20  weeks. 

In  addition,  one  female  and  one  male  (both  with  the  most  experience 
with  voice  recognition  equipment)  also  trained  the  recognizer  in  a  joint 
mode  on  all  240  utterances  in  which  5  training  passes  for  each  utterance 
were  provided  by  the  female  and  5  by  the  male.  The  validity  of  these 
training  passes  also  had  to  pass  the  2  out  of  3  criteria  (mentioned 
earlier)  for  both  the  male  and  female.  The  joint  reference  patterns  were 
then  recorded  on  a  tape  which  was  not  altered  for  the  next  20  weeks. 

V.  THE  VOCABULARY 

The  vocabulary  of  240  utterances  is  shown  on  the  data  sheet  in 
Appendix  B.  It  is  divided  into  groups  of  20.  Each  group  of  20  was 
balanced  with  two  1  syllable  utterances,  six  2  syllable  utterances,  four 
3  syllable  utterances,  four  4  syllable  utterances  and  four  utterances  of 
5  or  more  syllables.  These  were  selected  to  attempt  to  simulate  their 
frequency  of  use  in  a  command  center. 

VI.  PROCEDURE  AND  DATA  COLLECTED 

Each  week  for  20  weeks,  7  of  the  subjects  were  observed  and  measured 
on  recognition  performance  as  they  repeated  the  240  utterances  twice  per  the 


4 


instructions.  In  the  T600,  one  can  set  a  given  utterance's  prompt  to  null 
and  deactivate  that  utterance  and  any  beyond  it. 

Therefore,  each  week  a  subject's  voice  pattern  tape  was  read  into 
the  recognizer  memory  and  immediately  prompt  20  was  set  to  null  so  only 
voice  reference  patterns  <j>  through  19  were  active.  Then  the  subject 
repeated  these  utterances  once  and  then  went  through  them  again.  The  first 
20  utterances  were  therefore  being  tested  against  themselves.  Next  prompt 
#  20  was  reset  and  prompt  #  40  was  set  to  null  so  now  reference  patterns  <j> 
through  39  were  active.  Then  the  subject  repeated  the  second  20  utterances 
twice,  but  in  this  case  the  second  set  of  20  utterances  were  actually  being 
tested  against  all  40  active  voice  reference  patterns.  Then  when  the 
third  set  of  20  utterances  were  tested,  they  were  actually  tested  against 
an  active  vocabulary  of  €0.  Therefore,  the  vocabulary  was  opened  up  by 
groups  of  20  until  the  last  20  utterances  were  being  tested  against  an 
active  vocabulary  of  240  reference  patterns. 

Each  subject,  except  one  who  could  only  participate  for  8  weeks, 
provided  480  utterances  per  week  for  21  weeks.  All  eight  subjects  therefore 
contributed  a  total  of  74,400  tests  of  the  recognizer's  capability. 

In  addition,  the  male  and  female  who  had  made  the  joint  reference 
pattern  tape  did  "double"  duty  by  each  providing  another  480  utterances 
per  week  which  were  tested  against  the  patterns  on  the  tape  which  had 
their  joint  reference  patterns. 

These  two  subjects  contributed  20,160  utterances  in  this  portion  of 
the  experiment. 

After  20  weeks,  all  subjects  retrained  every  utterance  which  had  a 
misrecognition  at  any  time  during  the  first  20  weeks.  After  validation 
criteria  were  again  satisfied,  data  for  the  21st  week  was  collected. 


The  subjects  with  the  joint  tape  did  likewise.  Every  utterance 
which  previously  had  a  misrecognition  by  either  the  male  or  female  was 
retrained  therefore  by  both  subjects.  After  validation,  data  for  the  21st 
week  was  collected. 

All  data  was  collected  in  a  relatively  quiet  atmosphere  typical  of 
normal  operations  in  a  command  center  environment  or  office. 

VII.  RESULTS 

Results  for  the  longitudinal  portion  of  the  research  are  shown  in 
Figure  1.  An  analysis  of  variance  on  the  arcsin  transformation  of  the 
raw  data  over  each  week  by  each  subject  is  shown  in  Table  1.  Subject  4 
was  not  included  in  the  statistical  analysis  due  to  his  participating  in 
the  study  for  only  8  weeks.  The  statistical  results  of  Table  1  showed  an 
expected  difference  between  subjects  but  no  difference  in  performance  by 
week  over  the  21  weeks.  This  is  seen  by  observing  the  overall  average 
curve  (the  middle  one)  in  Figure  1  which  shows  less  than  a  1.7%  variation 
from  week  to  week  over  21  weeks.  Although  the  averages  for  the  males  and 
females  are  shown  in  this  figure  individually,  the  middle  average  line  is 
more  indicative  of  the  group's  performance  since  the  females'  line  is 
based  on  2  subjects  whereas  the  males'  is  based  on  six  subjects.  The 
overall  variability  from  week  to  week  was  large  enough  to  show  no  statistical 
difference  in  general  over  the  21  weeks  of  the  study.  As  can  be  seen,  per¬ 
formance  did  not  get  worse  and  worse  over  time  and  thus  the  initial  voice 
reference  patterns  provided  stable  performance  for  the  21  weeks. 

This  indicates  that  large  amounts  of  retraining  may  not  be  required 
over  extended  periods  of  time,  and  that  performance  can  remain  consistent. 

The  reader  will  recall  that  after  20  weeks,  all  utterances  which 
previously  had  any  misrecognitions  were  retrained.  One  might  have  thought 
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TABLE  I 


Analysis  of  Variance  on 
Subjects  over  21  Weeks 


Source  of 


Variation 

df 

MS 

F 

SUBJECTS 

6 

.4130 

82.60* 

WEEKS 

20 

.0056 

1.12 

RESIDUAL  ERROR 

120 

.0050 

TOTAL 

146 

*p  <  .01 

N.S.  =  not  significant 
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5  10  15 

WEEK 

Figure  1-  Error  rote  were  us  week* 
( individual  pattern*) 


this  would  help  and  subsequently  cause  a  large  improvement  in  performance 
during  the  21st  week.  However,  as  seen  in  Figure  1,  there  was  a  minimal 
improvement  but  nothing  statistically  significant  in  the  21st  week. 

One  will  notice  the  average  error  rate  did  increase  slightly  in 
week  8.  This  may  have  no  meaning  or  could  be  related  to  exams  which  would 
have  been  given  about  that  time.  Also  week  14  showed  a  fair  drop  in  error 
rate  and  may  have  been  related  to  the  fact  that  this  was  at  the  beginning 
of  a  new  academic  quarter  in  which  subjects  may  have  been  under  less 
pressure.  Since  these  observations  are  being  made  after  the  experiment, 
one  can  not  take  the  above  implications  as  true  fact.  However,  it  is 
something  to  consider  and  is  a  small  indication  of  possible  stress  factors 
at  work.  It  helps  to  point  out  that  much  remains  to  be  done  in  the  whole 
area  of  environmental  and  psychological  stress  effects  on  voice  recognition 
performance. 

Figure  2,  shows  the  results  of  the  investigation  into  vacabulary  size. 

It  can  be  seen  that  in  general,  voice  recognition  performance  remained 
relatively  constant  as  the  size  of  the  vocabulary  increased.  Statistical 
analysis  showed  that  the  error  rate  did  not  increase  as  a  function  of 
vocabulary  size  as  many  might  expect.  Although  this  writer  claims  no 
magic  credit  for  the  following,  a  lot  of  time  was  spent  developing  the 
vocabulary  used  here  which  may  be  typical  of  a  vocabulary  used  in  command 
center  operations.  In  addition,  each  group  of  20  utterances  was  balanced 
with  equal  numbers  of  utterances  of  1,2, 3, 4  and  5  (plus)  syllable  words. 

The  combination  of  vocabulary  choice  and  syllable  balancing  may  contribute 
to  performance  stability  as  a  function  of  vocabulary  size. 

As  a  result  of  using  different  numbers  of  syllables  in  the 
vocabulary,  it  was  also  possible  to  get  an  indication  of  how  well  utterances 
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Figure  2.  Mean  error  rate  versus  vocobulory  size. 


with  different  numbers  of  syllables  were  recognized.  This  is  shown  in 
Figure  3  which  illustrates  a  declining  error  rate  as  a  function  of  the 
number  of  syllables  in  the  utterance. 

Figure  4  shows  the  results  of  the  performance  for  each  subject  over 
all  21  weeks.  Subjects  1  (male)  and  6  (female)  were  the  participants 
in  the  joint  reference  pattern  investigation.  For  both  these  subjects, 
their  performance  dropped  about  .7%  when  using  the  joint  patterns  versus 
using  their  own  patterns.  It  is  also  interesting  to  note  that  subject  1, 
using  joint  patterns  of  female  subject  6,  performed  better  than  any  other 
subject  did  against  their  own  individual  reference  patterns.  Subject  4 
in  this  figure  is  the  one  who  only  participated  8  weeks. 

Figure  5,6,  and  7  simply  show  the  individual  subject's  error  rates 
by  week  for  the  entire  study.  Subject  1  and  6  in  Figure  5  are  the  same 
subjects  as  in  Figure  7  but  using  their  individual  voice  reference 
patterns  versus  their  combined  patterns. 
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Figure  4  s  Recognition  performance  by  subject  on  individual  patterns 
unless  noted. 
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Figure  5=  Performance  by  week  for  individual  subjects 
(own  voice  pattern). 
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Figure  7-  Performance  by  week  for  two  subjects  using 
combined  voice  reference  patterns. 
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VIII.  CONCLUSIONS 

The  main  purpose  of  this  study  was  to  determine  if  voice  recognition 
performance  could  be  stable  over  a  "long"  period  of  time.  It  is  apparent 
that  relative  stability  is  possible  and  one  may  not  have  to  expect  to  do 
a  lot  of  re-training  of  voice  patterns  as  time  progresses  over  many  months. 
As  a  matter  of  record,  this  writer  uses  two  year  old  voice  reference 
patterns  on  a  daily  basis  and  hardly  ever  re-trains  any  of  the  old  patterns. 

As  a  result  of  the  main  experiment,  it  was  also  possible  to  get  an 
idea  of  recognition  performance  as  a  function  of  vocabulary  size. 

Although  one  might  believe  that  the  error  rate  should  increase  dramatically 
as  larger  vocabularies  are  used,  the  current  results  indicate  that  this 
need  not  be  the  case.  Rather,  it  might  be  argued,  one  can  overcome  large 
vocabulary  difficulties  through  judicious  selection  of  the  vocabulary 
items  themselves,  as  others  have  suggested  in  past  discussions  at  con¬ 
ferences,  etc. 

Finally,  it  was  successfully  shown  that  a  male  and  a  remale  could 
combine  their  voice  reference  patterns  in  a  speaker-dependent  recognition 
system  and  still  achieve  high  recognition  performance.  This  has  many 
implications  for  use  in  emergency  situations  where  one  member  of  a  crew 
may  become  incapacitated  suddenly.  If  one  didn't  wish  to  have  joint 
patterns  for  all  utterances  as  done  in  this  study,  one  might  at  least 
have  joint  patterns  for  emergency  and  "STOP-ACTION"  types  of  commands  in 
the  vocabulary. 
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APPENDIX  A 


VOICE  STUDIES  AT  NPS 


This  project  is  one  of  several  voice  recognition  research  projects 

conducted  for/by  Professor  G.  K.  Poock  at  NPS  over  the  last  several  years. 

The  complete  list,  in  addition  to  this  report,  includes: 

Armstrong,  J.  W. ,  The  Effects  Of  Concurrent  Motor  Tasking  On  Performance 
Of  A  Voice  Recognition  System,  Masters  Thesis,  Naval  Postgraduate  School, 
Monterey,  1980. 

Batchellor,  M.  P.,  Investigation  Of  Parameters  Affecting  Voice  Recognition 
Systems  In  C3  Systems,  Masters  Thesis,  Naval  Postgraduate  School,  Monterey, 
1981. 

Bragaw,  P.  H. ,  Investigation  Of  Voice  Input  For  Constructing  Joint  Chiefs 
Of  Staff  Emergency  Action  Messages,  Masters  Thesis,  Naval  Postgraduate 
School,  Monterey,  1981. 

Jay,  G.  T.,  An  Experiment  In  Voice  Data  Entry  for  Imagery  Intelligence 
Reporting,  Masters  Thesis,  Naval  Postgraduate  School,  Monterey,  1981. 

McSorley,  W.  J.  Using  Voice  Recognition  Equipment  To  Run  The  Warfare 
Environmental  SimuTator  (WES),  Masters  Thesis,  Naval  Postgraduate 
School,  Monterey,  1981. 

Naval  Postgraduate  School  Report  NPS54-80-010,  The  Effects  Of  Certain 
Background  Noises  On  The  Performance  Of  A  Voice  Recognition  System, 
by  R.  Elster,  September  1980. 

Naval  Postgraduate  School  Report  NPS55-80-C16,  Experiments  With  Voice 
Input  For  Command  And  Control:  Using  Voice  Input  To  Operate  A  Distributed 
Computer  Network,  by  G.  K.  Poock,  April  1980. 

Naval  Postgraduate  School  Report  NPS55-81 -003 ,  Examination  Of  Voice 
Recognition  System  To  Function  In  A  Bilingual  Mode,  by  D.  E.  Neil  and 
T.  Andreason,  February  1981. 

Taggart,  J.  L.  and  Wolfe,  C.  D.,  Speech  Recognition  As  An  Input  Medium  For 
Freflight  In  The  P3C  Aircraft,  Masters  Thesis,  Naval  Postgraduate  School, 
Monterey,  1981. 
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appendix  b 


DATA  SHEET 


NAME 


LONGITUDINAL  STUDY 


DATE 


Is  this  data  based  on: 


L _ I  Your  voice  patterns? 

[  _  j  Yours  and  other's  patterns? 

(Whose?)  _ 

[  _^~1  Other's  patterns? 

(Whose?)  _ 


There  are  240  patterns  on  your  tape  (word  number  0  -  239  in  memory). 
INSTRUCTIONS: 


A.  Set  the  prompt  of  word  20  to  a  carriage  return. 

B.  Say  word  21  to  check  your  vocabulary  size  (it  shouldn ' t  work). 

C.  Say  utterances  0  thruugh  19. 

D.  Then  repeat  utterances  0  through  19.  We  are  getting  2  obser¬ 
vations  per  utterance. 

E.  If  the  machine  beeps,  write  "BEEP"  to  the  right  of  the  word  number. 

F.  If  the  machine  makes  a  mistake,  write  the  MISTAKEN  output  to  the 

right  of  the  word  number. 


Now  go  through  the  list  twice  on  the  next  page. 


WORD  // 


UTTERANCE 


0  ONE 

1  TWO 

2  YANKEE 

3  AIR  ROUTES 

4  GARY  POOCK 

5  LOAD  THE  GANN 

6  CARRIAGE  RETURN 

7  LOAD  THE  SERVER 

8  IRAN 

9  JAPAN 

10  SWEDEN 

11  EUROPE 

12  LOGIN  POOCK 

13  LEVEL  TWO 

14  ACCAT  TITLE 

15  STRAIT  OF  HORMUZ 

16  LOAD  GLD3 

17  CONNECT  TO  CHARLIE 

18  POOCK  NPS  PASSWORD 

19  CHANGE  DIRECTORY  TO  HUNTER 


20 


Now  SET  the  prompt  of  word  20  back  to  "three"  and  the  prompt  of  word  40 
to  carriage  return.  Check  vocabulary  size  by  saying  word  41.  It  should 
not  work.  Now  go  through  the  following  list  twice. 


20  THREE 

21  FOUR 

22  LOGOUT 

23  GRAPHICS  0 

24  RED  SPHERE 

25  STEAM  PLANT 

26  ZERO 

27  SEVEN 

28  NOVEMBER 

29  MOVE  IT  DOWN 

30  USE  THAT  ONE 

31  SPIROGRAPH 

32  CAPTAIN  EBBERT 

33  CLOSE  OUT  CHARLIE 

34  UP  IN  DETAIL 

35  UNITED  STATES 

36  LEVEL  TWO  VIEWER 

37  NORTH  ATLANTIC  MAP 

38  GENISCO  ZERO  PARAMETERS 


39 


MEDITERRANEAN  MAP 


Now  set  the  prompt  of  word  40  back  to  "FIVE"  and  set  the  prompt  of  word 
60  to  carriage  return.  CHECK  vocabulary  size  by  saying  word  61.  It 
should  not  work.  Now  go  through  the  following  list  twice. 


40  FIVE 

41  SIX 

42  ALPHA 

43  BRAVO 

44  CHARLIE 

45  DELTA 

46  ECHO 

47  FOXTROT 

48  JULIETT 

49  ROMEO 

50  MOVE  IT  LEFT 

51  SIERRA 

52  SAN  FRANCISCO 

53  APPLICATION 

54  ENGINEERING 

55  HUMAN  FACTORS 

56  VOICE  TECHNOLOGY 

57  CENTRAL  EXPRESSWAY 

58  RUSSIAN  VERSION  OF  HORMUZ 

59  FILE  TRANSFER  PROTOCOL 
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Now  Now  set  the  prompt  of  word  60  back  to  "EIGHT"  and  set  the  prompt 
of  word  80  to  carriage  return.  Check  vocabulary  size  by  saying  word 
81.  It  should  not  work.  Now  go  through  the  following  list  twice. 


60  EIGHT 

61  NINE 

62  HOTEL 

63  INDIA 

64  KILO 

65  LIMA 

66  OSCAR 

67  POPPA 

68  MOVE  IT  RIGHT 

69  UNIFORM 

70  VIETNAM 

71  KOREA 

72  ADVISORY 

73  INTERACTIVE 

74  BUSINESS  MEETING 

75  CONTINUOUS 

76  SPEECH  RECOGNITION 

77  CONTINUOUS  SPEECH 

78  EFFICIENT  TRANSMISSION 

79  SYSTEM  INTEGRATION 


23 


Now  set  the  prompt  of  word  80  back  to  "GOLF"  and  set  the  prompt  of  word 
100  to  carriage  return.  Check  vocabulary  size  by  saying  word  101.  It 
should  not  work.  Now  go  through  the  following  list  twice. 


80  GOLF 

81  MIKE 

82  QUEBEC 

83  TANGO 

84  VICTOR 

85  WHISKEY 

86  XRAY 

87  ZULU 

88  MOVE  IT  UP 

89  BANGLADESH 

90  TOKYO 

91  HOLLISTER 

92  DOWN  IN  DETAIL 

93  CORPORATION 

94  CRITERIA 

95  ADVANTAGES 

96  SUITABILITY 

97  RADIOLOGY 

98  IDENTIFICATION 

99  AUTOMATIC  RECOGNITION 
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Now  set  the  prompt  or  word  100  back  to  "COURSE"  and  set  the  prompt  of 
word  120  to  carriage  return.  Check  vocabulary  size  by  saying  word  121. 
It  should  not  work.  Now  go  through  the  following  list  twice. 


100  COURSE 

101  SPEED 

102  COMMAND 

103  ATTACK 

104  BINGO 

105  REPORT 

106  PROCEED 

107  STATION 

108  ALTITUDE 

109  RECOVER 

110  RELOCATE 

111  DESIGNATE 

112  AVAILABLE 

113  PLOT  ESM 

114  TRACK  ENEMY 

115  DESIGNATE  TRACK 

116  COMMAND  AND  CONTROL 

117  PROBABILITY 

118  ENEMY  DETECTION 

119  PROBABILITY  OF  DETECTION 
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Now  set  the  prompt  of  word  120  back  to  "LAUNCH”  and  set  the  prompt  of 
word  140  to  carriage  return.  Check  vocabulary  size  by  saying  word  141. 
It  should  not  work.  Now  go  through  the  following  list  twice. 


h 

i 

l 

k; 


1 


K 

f 


120  LAUNCH 

121  FIRE 

122  CANCEL 

123  MESSAGE 

124  BEARING 

125  LABEL 

126  ORDERS 

127  COPY 

128  SATELLITE 

129  ENVELOPE 

130  NEGATIVE 

131  CORRELATE 

132  COMBINATION 

133  SENSOR  DELAY 

134  MANEUVER  DELAY 

135  ALABAMA 

136  TASK  FORCE  COMMANDER 

137  NORTH  CAROLINA 

138  PROCEED  TO  NEW  DELHI 

139  PLACE  A  CIRCLE  ON  MOSCOW 


Now  set  the  prompt  of  word  140  back  to  "TIME"  and  set  the  prompt  of  word 
160  to  carriage  return.  Check  vocabulary  size  by  saying  word  161.  It 
should  not  work.  Now  go  through  the  following  list  twice. 


140 

TIME 

141 

SHOOT 

142 

SURFACE 

143 

REFUEL 

144 

MINEFIELD 

145 

DISTANCE 

146 

SHORE  BASED 

147 

CONTACT 

148 

EXECUTE 

149 

SUBMARINE 

150 

ENEMY 

151 

ORDER  NAME 

152 

CONNECTICUT 

153 

INDIANA 

154 

OKLAHOMA 

155 

PENNSYLVANIA 

156 

CALIFORNIA 

157 

SOUTH  DAKOTA 

158 

PLACE  A  MARKER 

ON  PARIS 

159 

BINGO  ALL  CRAFT 

IMMEDIATELY 
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Now  set  the  prompt  of  word  160  back  to  "MAP"  and  set  the  prompt  of 
word  180  to  carriage  return.  Check  vocabulary  size  by  saying  word  181. 
It  should  not  work.  Now  go  through  the  following  list  twice. 

160  MAP 

161  GRID 

162  NEUTRAL 

163  MISSILE 

164  SENSOR 

165  ADAK 

166  STOCKTON 

167  NEW  YORK 

168  AIR  FIELD  NAME 

169  TRACK  UNKNOWN 

170  TRACK  FRIENDLY 

171  TRACK  NEUTRAL 

172  BEARING  AND  DISTANCE 

173  LOUISIANA 

174  MINNESOTA 

175  COLORADO 

176  EISENHOWER 

177  NEW  MEXICO 

178  RELOCATE  THE  SUNFISH 

179  REFUEL  THE  CONNIE 


28 


! 


Now  set  the  prompt  of  word  180  back  to  "TAKE"  and  set  the  prompt  of 
word  200  to  carriage  return.  Check  vocabulary  size  by  saying  word  201. 
It  should  not  work.  Now  go  through  the  following  list  twice. 


180  TAKE 

181  PLACE 

182  GEORGIA 

183  VERMONT 

184  TEXAS 

185  DANIELS 

186  UTAH 

187  PLATFORM 

188  LATITUDE 

189  LONGITUDE 

190  OHIO 

191  TORPEDO 

192  FLIGHT  CONTROLLER 

193  TRANS  WORLD  AIRLINES 

194  PANGO  PANGO 

195  KEEP  ON  STATION 

196  LAY  A  BARRIER 

197  GROUND  CONTROL  APPROACH 

198  ATTACK  BARRIER  TARGET 


199 


ATLANTIC  DATA  BASE 


Now  set  the  prompt  of  word  200  back  to  "SCOPE"  and  set  the  prompt  of 
word  220  to  carriage  return.  Check  vocabulary  size  by  saying  word  221 
It  should  not  work.  Now  go  through  the  following  list  twice. 


200  SCOPE 

201  DROP 

202  BANGKOK 

203  BOMBAY 

204  BRISBANE 

205  CANTON 

206  ANTWERP 

207  AFRICA 

208  ARKANSAS 

209  SAIGON 

210  USER'S  GUIDE 

211  KITTY  HAWK 

212  ACAPULCO 

213  VLADIVOSTOK 

214  YOKOHAMA 

215  SEA  OF  JAPAN 

216  DIEGO  GARCIA 

217  INDONESIA 

218  PACIFIC  DATA  BASE 


219 


ARABIAN  TANKER 


Now  set  the  prompt  of  word  220  back  to  "MAINE"  Now  go  through  the 
following  list  twice. 


220  MAINE 

221  SAVE 

222  PORTLAND 

223  RANGOON 

224  ASPRO 

225  KIEV 

226  RED  FOX 

227  NAPLES 

228  BLUE  FORCE  ONE 

229  CALCUTTA 

230  BALTIMORE 

231  WYOMING 

232  SEVASTOPOL 

233  HONOLULU 

234  KRENOMETR 

235  JOHN  KENNEDY 

236  PLOT  ALL  SUBMARINES 

237  UNITED  AIR  LINES 

238  IBERIAN  CARRIER 

239  WEST  GERMAN  TORPEDO 


Now  RETURN  this  data  sheet  to  GARY  POOCK. 
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